Three-dimensional imaging device, image processing device, image processing method, and program

ABSTRACT

A 3D image capture device includes: a light transmitting section  2  with m transmitting areas (where m is an integer and m≧2) that have different spectral transmittance characteristics; and an image sensor  1  which is arranged to receive light rays that have been transmitted through the m transmitting areas and in which unit elements are arranged. Each unit element includes n photosensitive cells (where n is an integer and n≧m) and n transmitting filters with mutually different spectral transmittance characteristics which are arranged so as to face the n photosensitive cells. The image capture device further includes an image processing section which modifies an n×m matrix that is defined by the respective spectral transmittance characteristics of the m transmitting areas and the n transmitting filters by changing the angle between at least two out of m column vectors of the n×m matrix and which generates multi-viewpoint images using the modified n×m matrix.

TECHNICAL FIELD

The present application relates to a single-lens 3D image capturingtechnology for generating multiple images with parallax.

BACKGROUND ART

Recently, the performance and functionality of digital cameras anddigital movie cameras that use some solid-state image sensor such as aCCD and a CMOS (which will be sometimes simply referred to herein as an“image sensor”) have been enhanced to an astonishing degree. Inparticular, the size of a pixel structure for use in a solid-state imagesensor has been further reduced these days thanks to rapid developmentof semiconductor device processing technologies, thus getting an evengreater number of pixels and drivers integrated together in asolid-state image sensor. As a result, the resolution of an image sensorhas lately increased rapidly from around one million pixels to tenmillion or more pixels in a matter of few years. On top of that, thequality of an image captured has also been improved significantly aswell. As for display devices, on the other hand, LCD and plasma displayswith a reduced depth now provide high-resolution and high-contrastimages, thus realizing high performance without taking up too muchspace. And such video quality improvement trends are now spreading from2D images to 3D images. In fact, 3D display devices that achieve highimage quality although they require the viewer to wear a pair ofpolarization glasses have been developed just recently.

As for the 3D image capturing technology, a typical 3D image capturedevice with a simple arrangement uses an image capturing system with twocameras to capture a right-eye image and a left-eye image. According tothe so-called “two-lens image capturing” technique, however, two camerasneed to be used, thus increasing not only the overall size of the imagecapture device but also the manufacturing cost as well. To overcome sucha problem, methods for capturing multiple images with parallax (whichwill be sometimes referred to herein as a “multi-viewpoint image”) byusing a single camera have been researched and developed. Such a methodis called a “single-lens image capturing method”. For example, PatentDocument No. 1 discloses a technique for obtaining two images withparallax at the same time using color filters. FIG. 16 schematicallyillustrates an image capturing system that adopts such a technique. Theimage capturing system that uses that technique includes a lens 3, alens diaphragm 19, a light beam confining plate 20 with two colorfilters 20 a and 20 b that have mutually different transmissionwavelength ranges, and a photosensitive film 21. In this case, the colorfilters 20 a and 20 b may be filters that transmit red- and blue-basedlight rays, respectively.

In such an arrangement, the incoming light passes through the lens 3,the lens diaphragm 19 and the light beam confining plate 20 and producesan image on the photosensitive film 21. In the meantime, only red- andblue-based light rays are respectively transmitted through the two colorfilters 20 a and 20 b of the light beam confining plate 20. As a result,a magenta-based color image is produced on the photosensitive film 21 bythe light rays that have been transmitted through the two color filters.In this case, since the color filters 20 a and 20 b are arranged atmutually different positions, the image produced on the photosensitivefilm 21 comes to have parallax. Thus, if a photograph is developed withthe photosensitive film and viewed with a pair of glasses, in which redand blue films are attached to its right- and left-eye lenses, theviewer can view an image with depth. In this manner, according to thetechnique disclosed in Patent Document No. 1, multi-viewpoint images canbe produced using the two color filters.

According to the technique disclosed in Patent Document No. 1, the lightrays are imaged on the photosensitive film, thereby producing multipleimages with parallax there. Meanwhile, Patent Document No. 2 discloses atechnique for producing images with parallax by transforming incominglight into electrical signals. FIG. 17 schematically illustrates a lightbeam confining plate 22 according to such a technique. Specificallyaccording to that technique, a light beam confining plate 22, which hasa red ray transmitting R area 22R, a green ray transmitting G area 22Gand a blue ray transmitting B area 22B, is arranged on a plane thatintersects with the optical axis of the imaging optical system at rightangles. And by getting the light rays that have been transmitted throughthose areas received by a color image sensor that has red-, green- andblue-ray-receiving R, G and B pixels, an image is generated based on thelight rays that have been transmitted through those areas.

Patent Document No. 3 also discloses a technique for obtaining imageswith parallax using a similar configuration to the one illustrated inFIG. 17. FIG. 18 schematically illustrates a light beam confining plate23 as disclosed in Patent Document No. 3. According to that technique,by making the incoming light pass through R, G and B areas 23R, 23G and23B of the light beam confining plate 23, multiple images with parallaxcan also be produced.

Patent Document No. 4 also discloses a technique for generating multipleimages with parallax using a pair of filters with mutually differentcolors, which are arranged symmetrically to each other with respect toan optical axis. By using red and blue filters as the pair of filters,an R pixel that senses a red ray observes the light that has beentransmitted through the red filter, while a B pixel that senses a blueray observes the light that has been transmitted through the bluefilter. Since the red and blue filters are arranged at two differentpositions, the light received by the R pixel and the light received bythe B pixel have come from mutually different directions. Consequently,the image observed by the R pixel and the image observed by the B pixelare ones viewed from two different viewpoints. By defining correspondingpoints between those images on a pixel-by-pixel basis, the magnitude ofparallax can be calculated. And based on the magnitude of parallaxcalculated and information about the focal length of the camera, thedistance from the camera to the subject can be obtained.

Patent Document No. 5 discloses a technique for obtaining informationabout a subject distance based on two images that have been generatedusing either a diaphragm to which two color filters with mutuallydifferent aperture sizes (e.g., red and blue color filters) are attachedor a diaphragm to which two color filters in two different colors areattached horizontally symmetrically with respect to the optical axis.According to such a technique, if light rays that have been transmittedthrough the red and blue color filters with mutually different aperturesizes are observed, the degrees of blur observed vary from one color toanother. That is why the degrees of blur of the two images that areassociated with the red and blue color filters vary according to thesubject distance. By defining corresponding points with respect to thoseimages and comparing their degrees of blur to each other, informationabout the distance from the camera to the subject can be obtained. Onthe other hand, if light rays that have been transmitted through twocolor filters in two different colors that are attached horizontallysymmetrically with respect to the optical axis are observed, thedirection from which the light observed has come changes from one colorto another. As a result, two images that are associated with the red andblue color filters become images with parallax. And by definingcorresponding points with respect to those images and calculating thedistance between those corresponding points, information about thedistance from the camera to the subject can be obtained.

According to the techniques disclosed in Patent Documents Nos. 1 to 5mentioned above, images with parallax can be produced by arranging RGBcolor filters on a light beam confining plate or a diaphragm. However,since the RGB based color filters are used, the percentage of theincoming light that can be used decreases significantly. In addition, toincrease the effect of parallax, those color filters should be arrangedat distant positions and should have decreased areas. In that case,however, the percentage of the incoming light that can be used furtherdecreases.

Unlike these techniques, Patent Document No. 6 discloses a technique forobtaining multiple images with parallax and a normal image that is freefrom the light quantity problem by using a diaphragm in which RGB colorfilters are arranged. According to that technique, when the diaphragm isclosed, only the light rays that have been transmitted through the RGBcolor filters are received. On the other hand, when the diaphragm isopened, the RGB color filter areas are outside of the optical path, andtherefore, the incoming light can be received entirely. Consequently,multi-viewpoint images can be obtained when the diaphragm is closed anda normal image that uses the incoming light highly efficiently can beobtained when the diaphragm is opened.

CITATION LIST Patent Literature

-   -   Patent Document No. 1: Japanese Laid-Open Patent Publication No.        2-171737    -   Patent Document No. 2: Japanese Laid-Open Patent Publication No.        2002-344999    -   Patent Document No. 3: Japanese Laid-Open Patent Publication No.        2009-276294    -   Patent Document No. 4: Japanese Laid-Open Patent Publication No.        2010-38788    -   Patent Document No. 5: Japanese Laid-Open Patent Publication No.        2010-79298    -   Patent Document No. 6: Japanese Laid-Open Patent Publication No.        2003-134533

SUMMARY OF INVENTION Technical Problem

According to any of the techniques disclosed in Patent Documents Nos. 1to 5, multi-viewpoint images can be certainly obtained, but the quantityof the light received by the image sensor is much smaller than usualbecause primary color (RGB) based color filters are used. On the otherhand, according to the technique disclosed in Patent Document No. 6, anormal image that uses the incoming light highly efficiently can beobtained by using a mechanism that removes a color filter from theoptical path by mechanical driving. Even with that technique, however,primary color based color filters are also used to obtainmulti-viewpoint images. Consequently, the multi-viewpoint images cannotbe obtained with the incoming light used sufficiently efficiently. Ontop of that, according to such a technique, the overall size of thedevice increases too much and the manufacturing cost becomes too high.

An embodiment of the present invention provides an image capturingtechnique for obtaining multi-viewpoint images with the incoming lightused highly efficiently without making any mechanical driving.

Solution to Problem

To overcome these problems, a 3D image capture device as an embodimentof the present invention includes: a light transmitting section with mtransmitting areas (where m is an integer that is equal to or greaterthan two) that have mutually different spectral transmittancecharacteristics; and an image sensor which is arranged to receive lightrays that have been transmitted through the m transmitting areas and inwhich a plurality of unit elements are arranged. Each unit elementincludes n photosensitive cells (where n is an integer that is equal toor greater than m) and n transmitting filters with mutually differentspectral transmittance characteristics which are arranged so as to facethe n photosensitive cells. The image capture device further includes:an imaging section which produces an image on the imaging area of theimage sensor; and an image processing section which modifies an n×mmatrix that is defined by the respective spectral transmittancecharacteristics of the m transmitting areas and the n transmittingfilters by changing the angle between at least two out of m columnvectors of the n×m matrix and which generates multi-viewpoint imagesrepresented by light rays that have been incident on at least two of them transmitting areas based on the modified n×m matrix and nphotoelectrically converted signals supplied from the n photosensitivecells.

These general and particular embodiments can be implemented as a system,a method, a computer program or a combination thereof.

Advantageous Effects of Invention

According to an embodiment of the present invention, multi-viewpointimages can be obtained without making any mechanical driving and withthe light used more efficiently than ever.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A block diagram illustrating an overall configuration for a firstembodiment.

FIG. 2 A schematic representation generally illustrating the relativearrangement of a light-transmitting plate, an optical system and animage sensor according to the first embodiment.

FIG. 3 A view illustrating an arrangement of transmitting areas on alight-transmitting plate according to the first embodiment.

FIG. 4 A view illustrating a basic arrangement of transmitting filtersin the image sensor of the first embodiment.

FIG. 5 A graph showing exemplary spectral transmittances of thetransmitting filters of the light-transmitting plate.

FIG. 6 A graph showing exemplary spectral transmittances of thetransmitting filters of the image sensor.

FIG. 7 A graph showing other exemplary spectral transmittances of thetransmitting filters of the light-transmitting plate.

FIG. 8 A graph showing other exemplary spectral transmittances of thetransmitting filters of the image sensor.

FIG. 9 Schematically illustrates an example of two vectors.

FIG. 10 Shows examples of an input image and multi-viewpoint images.

FIG. 11 A graph showing how the determinant value, the correlationvalue, and the degree of confidence change with θ.

FIG. 12 A flowchart showing an exemplary flow of processing steps to becarried out according to the first embodiment.

FIG. 13 Shows examples of multi-viewpoint images generated as a resultof the processing of the first embodiment.

FIG. 14 A view illustrating an example of a light-transmitting platewith m transmitting areas.

FIG. 15 A view illustrating an exemplary arrangement of n transmittingfilters in each unit element of the image sensor.

FIG. 16 A view illustrating the arrangement of an image capturing systemaccording to Patent Document No. 1.

FIG. 17 A view illustrating the appearance of a light beam confiningplate according to Patent Document No. 2.

FIG. 18 A view illustrating the appearance of a light beam confiningplate according to Patent Document No. 3.

DESCRIPTION OF EMBODIMENTS

(1) A 3D image capture device as an embodiment of the present inventionincludes: a light transmitting section with m transmitting areas (wherem is an integer that is equal to or greater than two) that have mutuallydifferent spectral transmittance characteristics; and an image sensorwhich is arranged to receive light rays that have been transmittedthrough the m transmitting areas and in which a plurality of unitelements are arranged. Each unit element includes n photosensitive cells(where n is an integer that is equal to or greater than m) and ntransmitting filters with mutually different spectral transmittancecharacteristics which are arranged so as to face the n photosensitivecells. The image capture device further includes: an imaging sectionwhich produces an image on the imaging area of the image sensor; and animage processing section which modifies an n×m matrix that is defined bythe respective spectral transmittance characteristics of the mtransmitting areas and the n transmitting filters by changing the anglebetween at least two out of m column vectors of the n×m matrix and whichgenerates multi-viewpoint images represented by light rays that havebeen incident on at least two of the m transmitting areas based on themodified n×m matrix and n photoelectrically converted signals suppliedfrom the photosensitive cells.

(2) In one embodiment, the image processing section increases thedeterminant of the n×m matrix by increasing the angle between at leasttwo of the m column vectors.

(3) In one embodiment of the 3D image capture device of (1) or (2), theimage processing section changes the angle so that a value representingcorrelation between the multi-viewpoint images becomes smaller than apredetermined threshold value.

(4) In one embodiment of the 3D image capture device of one of (1) to(3), the image processing section increases the angle so that thedeterminant of the n×m matrix becomes greater than a predeterminedthreshold value.

(5) In one embodiment of the 3D image capture device of one of (1) to(4), the image processing section increases the angle between two columnvectors that is smaller than the angle formed by any other pair of the mcolumn vectors.

(6) In one embodiment of the 3D image capture device of one of (1) to(5), the image processing section obtains the angle between twoarbitrary ones of the m column vectors and increases the anglessequentially by beginning with the smallest one, thereby increasing thedeterminant of the n×m matrix.

(7) In one embodiment of the 3D image capture device of one of (1) to(6), m=2 and n=2.

(8) An image processor as an embodiment of the present inventiongenerates multi-viewpoint images based on a signal that has beenobtained by a 3D image capture device. The 3D image capture deviceincludes: a light transmitting section with m transmitting areas (wherem is an integer that is equal to or greater than two) that have mutuallydifferent spectral transmittance characteristics; and an image sensorwhich is arranged to receive light rays that have been transmittedthrough the m transmitting areas and in which a plurality of unitelements are arranged. Each unit element includes n photosensitive cells(where n is an integer that is equal to or greater than m) and ntransmitting filters with mutually different spectral transmittancecharacteristics which are arranged so as to face the n photosensitivecells. The image capture device further includes an imaging sectionwhich produces an image on the imaging area of the image sensor. Theimage processor modifies an n×m matrix that is defined by the respectivespectral transmittance characteristics of the m transmitting areas andthe n transmitting filters by changing the angle between at least twoout of m column vectors of the n×m matrix and generates multi-viewpointimages represented by light rays that have been incident on at least twoof the m transmitting areas based on the modified n×m matrix and nphotoelectrically converted signals supplied from the n photosensitivecells.

(9) An image processing method as an embodiment of the present inventionis designed to generate multi-viewpoint images based on a signal thathas been obtained by a 3D image capture device. The device includes: alight transmitting section with m transmitting areas (where m is aninteger that is equal to or greater than two) that have mutuallydifferent spectral transmittance characteristics; and an image sensorwhich is arranged to receive light rays that have been transmittedthrough the m transmitting areas and in which a plurality of unitelements are arranged. Each unit element includes n photosensitive cells(where n is an integer that is equal to or greater than m) and ntransmitting filters with mutually different spectral transmittancecharacteristics which are arranged so as to face the n photosensitivecells. The image capture device further includes an imaging sectionwhich produces an image on the imaging area of the image sensor. Theimage processing method comprises the steps of: modifying an n×m matrixthat is defined by the respective spectral transmittance characteristicsof the m transmitting areas and the n transmitting filters by changingthe angle between at least two out of m column vectors of the n×mmatrix; and generating multi-viewpoint images represented by light raysthat have been incident on at least two of the m transmitting areasbased on the modified n×m matrix and n photoelectrically convertedsignals supplied from the photosensitive cells.

(10) An image processing program as an embodiment of the presentinvention is designed to generate multi-viewpoint images based on asignal that has been obtained by a 3D image capture device. The 3D imagecapture device includes: a light transmitting section with mtransmitting areas (where m is an integer that is equal to or greaterthan two) that have mutually different spectral transmittancecharacteristics; and an image sensor which is arranged to receive lightrays that have been transmitted through the m transmitting areas and inwhich a plurality of unit elements are arranged. Each unit elementincludes n photosensitive cells (where n is an integer that is equal toor greater than m) and n transmitting filters with mutually differentspectral transmittance characteristics which are arranged so as to facethe n photosensitive cells. The 3D image capture device further includesan imaging section which produces an image on the imaging area of theimage sensor. The image processing program is defined so as to make acomputer perform the steps of: modifying an n×m matrix that is definedby the respective spectral transmittance characteristics of the mtransmitting areas and the n transmitting filters by changing the anglebetween at least two out of m column vectors of the n×m matrix; andgenerating multi-viewpoint images represented by light rays that havebeen incident on at least two of the m transmitting areas based on themodified n×m matrix and n photoelectrically converted signals suppliedfrom the n photosensitive cells.

Hereinafter, specific embodiments of the present invention will bedescribed with reference to the accompanying drawings. In the followingdescription, any element shown in multiple drawings and havingsubstantially the same function will be identified by the same referencenumeral. It should be noted that a signal or information representing animage will be sometimes referred to herein as just an “image”.

Embodiment 1

FIG. 1 is a block diagram illustrating an overall configuration for animage capture device as a first embodiment of the present invention. Theimage capture device of this embodiment is a digital electronic cameraand includes an image capturing section 100 and a signal processingsection 200 that receives a signal from the image capturing section 100and outputs a signal representing an image (i.e., an image signal).

The image capturing section 100 includes a solid-state image sensor 1(which will be simply referred to herein as an “image sensor”) with anumber of photosensitive cells (pixels) that are arranged on its imagingarea, a light-transmitting plate (light-transmitting section) 2, whichhas two transmitting areas, of which the transmittances have mutuallydifferent wavelength dependences (i.e., different spectraltransmittances), an optical lens 3 for producing an image on the imagingarea of the image sensor 1, and an infrared cut filter 4. The imagecapturing section 100 further includes a signal generating and receivingsection 5, which not only generates a fundamental signal to drive theimage sensor 1 but also receives the output signal of the image sensor 1and sends it to the signal processing section 200, and a sensor drivingsection 6 for driving the image sensor 1 in accordance with thefundamental signal generated by the signal generating and receivingsection 5. The image sensor 1 is typically a CCD or CMOS sensor, whichmay be fabricated by known semiconductor device processing technologies.The signal generating and receiving section 5 and the sensor drivingsection 30 may be implemented as an LSI such as a CCD driver.

The signal processing section 200 includes an image signal generatingsection 7 for generating an image signal by processing the signalsupplied from the image capturing section 100, a memory 30 for storingvarious kinds of data for use to generate the image signal, and aninterface (I/F) section 8 for sending out the image signal thusgenerated to an external device. The image signal generating section 7may be a combination of a hardware component such as a known digitalsignal processor (DSP) and a software program for use to perform imageprocessing involving the image signal generation. The memory 30 may be aDRAM, for example. And the memory 30 not only stores the signal suppliedfrom the image capturing section 100 but also temporarily retains theimage data that has been generated by the image signal generatingsection 7 or compressed image data. These image data are then output toeither a storage medium or a display section (neither is shown) by wayof the interface section 8.

The image capture device of this embodiment actually further includes anelectronic shutter, a viewfinder, a power supply (or battery), aflashlight and other known components. However, description thereof willbe omitted herein because none of them are essential components thatwould make it difficult to understand how the present invention worksunless they were described in detail.

Next, the configuration of the image capturing section 100 will bedescribed in further detail with reference to FIGS. 2 through 4.

FIG. 2 schematically illustrates the relative arrangement of thelight-transmitting plate 2, the optical lens 3 and the image sensor 1 inthe image capturing section 100. It should be noted that illustration ofthe other elements is omitted in FIG. 2. The light-transmitting plate 2has two transmitting areas C1 and C2 that have mutually differentspectral transmittances and transmits the incoming light. The opticallens 3 is a known lens and condenses the light that has been transmittedthrough the light-transmitting plate 2, thereby imaging the light on theimaging area 1 a of the image sensor 1. The rest of thelight-transmitting plate 2 other than the transmitting areas C1 and C2is made of an opaque member, and this light-transmitting plate 2 isconfigured to prevent the incoming light from being transmitted throughthe area other than the transmitting areas C1 and C2. In the coordinatesystem used in the following description, the direction that points fromthe area C1 toward the area C2 will be referred to herein as “xdirection” and the direction that is defined perpendicularly to the xdirection on a plane parallel to the imaging area 1 a will be referredto herein as “y direction”. It should be noted that the arrangement ofthe respective members shown in FIG. 2 is only an example of the presentinvention. And the present invention is in no way limited to thatspecific embodiment. Alternatively, as long as an image can be producedon the imaging area 1 a, the lens 3 may be arranged more distant fromthe image sensor 1 than the light-transmitting plate 2 is. Stillalternatively, the lens 3 and the light-transmitting plate 2 may also beimplemented as a single optical element.

FIG. 3 is a front view of the light-transmitting plate 2 of thisembodiment. The light-transmitting plate 2, as well as the lens 3, has acircular shape in this embodiment but may also have any other shape. Ineach of the areas C1 and C2, arranged is a transmitting filter thattransmits at least partially a light ray falling within an arbitrarywavelength range included in the wavelength range of visible radiationW. Each of those transmitting filters transmits a light ray fallingwithin an arbitrary wavelength range included in the wavelength range ofthe visible radiation. However, since their spectral transmittances aredifferent, the light transmitted will have different brightness (orluminance) values depending on whether the light has been transmittedthrough the area C1 or the area C2. The spectral transmittances of therespective transmitting areas will be described in detail later. As longas each transmitting filter has the function of transmitting the lightat an intended transmittance, the filter may be made of any material.For example, the transmitting filters may be made of glass, plastic,cellophane or any other suitable material. Although transmitting filterswith mutually different spectral transmittances are arranged in thisembodiment in the transmitting areas C1 and C2, those areas may be madeof any other member as long as the member has the intended spectraltransmittance. For instance, if one of the two transmitting areas needsto be transparent, then that area may be replaced with the air.

The areas C1 and C2 are arranged with a certain gap L left in the xdirection. The gap L is determined by the size of the lens 3 so that theimage obtained will have appropriate parallax, and may be set to bewithin the range of a few millimeters to several centimeters, forexample. As shown in FIG. 3, the transmitting areas C1 and C2 aresuitably arranged horizontally symmetrically (i.e., in the x direction)with respect to the optical axis and have the same area. If such anarrangement is adopted, the quantities of the light rays to be incidenton the left and right areas C1 and C2 become substantially equal to eachother. It should be noted that the arrangement of the transmitting areasC1 and C2 does not have to be the one shown in FIG. 3 but may also bedetermined appropriately according to the intended use. For example, ifinformation about vertical parallax (i.e., in the y direction) needs tobe obtained, then the transmitting areas C1 and C2 may be arranged inthe y direction. Also, if the respective transmittances of thetransmitting areas C1 and C2 are significantly different from eachother, then the pixel values to be observed will also be quitedifferent. As a result, two images to be obtained will have differentbrightness values. That is why if there is a significant difference intransmittance between those transmitting areas C1 and C2, the planarareas of those areas C1 and C2 may be adjusted so that two images to beobtained will have close brightness values.

On the imaging area 1 a of the image sensor 1 shown in FIG. 2, there isan array of photosensitive cells that are arranged two-dimensionally andan array of transmitting filters that are arranged to face thosephotosensitive cells in the array. The array of photosensitive cells andthe array of transmitting filters consist of multiple unit elements. Inthis embodiment, each unit element includes two photosensitive cells andtwo associated transmitting filters that face them. Each of thosephotosensitive cells is typically a photodiode, which performsphotoelectric conversion and outputs an electrical signal representingthe quantity of the light received (which will be referred to herein asa “photoelectrically converted signal” or a “pixel signal”). On theother hand, each transmitting filter may be made of a known pigment or astack of dielectric materials and is designed so as to transmit at leasta part of a light ray with an arbitrary wavelength falling within thevisible radiation wavelength range.

FIG. 4 is a top view schematically illustrating a portion of the arrayof transmitting filters according to this embodiment. As shown in FIG.4, a lot of transmitting filters 110 are arranged in columns and rows onthe imaging area 1 a. As described above, each unit element includes twotransmitting filters 110 that are arranged close to each other and twophotosensitive cells 120 that face them. The two transmitting filters D1and D2 that are included in each unit element both transmit a light raywith an arbitrary wavelength falling within the visible radiationwavelength range but their transmittances have mutually differentwavelength dependences.

In the example illustrated in FIG. 4, two photosensitive cells arearranged horizontally (i.e., in the x direction). However, thephotosensitive cells 120 may also be arranged in any other pattern. Forexample, the photosensitive cells may be arranged vertically (i.e., inthe y direction) or obliquely. Furthermore, the number of photosensitivecells 120 included in a single unit element does not have to be two butmay also be three or more. Moreover, the photosensitive cells 120 andthe transmitting filters 110 do not have to be arranged in the x and ydirections but may also be arranged obliquely with respect to the x andy directions.

Next, exemplary spectral transmittances of the transmitting areas C1 andC2 and the transmitting filters D1 and D2 according to this embodimentwill be described briefly. FIG. 5 shows exemplary spectraltransmittances of the transmitting areas C1 and C2 of thelight-transmitting plate 2, and FIG. 6 shows exemplary spectraltransmittances of the transmitting filters D1 and D2 of the image sensor1. In this example, the spectral transmittance Tc1 of the area C1 isrepresented by a waveform similar to a rectangular wave, of which thespectral transmittance becomes almost 100% in the visible radiationwavelength range (i.e., from approximately 400 nm to approximately 700nm). On the other hand, the spectral transmittance Tc2 of the area C2 isrepresented by a waveform similar to a cos curve in the visibleradiation wavelength range. Meanwhile, the spectral transmittance Td1 ofthe transmitting filter D1 is represented by a waveform similar to arectangular wave, while the spectral transmittance Td2 of thetransmitting filter D2 is represented by a waveform similar to a sincurve. In this example, as for Tc1 and Td1, the transmittance is almost100% at any wavelength, and therefore, light hardly attenuates and goodsensitivity characteristic is realized.

The filters do not have to be designed as shown in FIGS. 5 and 6 but mayalso be designed to make Tc1 and Tc2 different from each other and alsomake Td1 and Td2 different from each other. For example, the respectivetransmitting areas and transmitting filters may be designed so that Tc1,Tc2, Td1 and Td2 are represented by waveforms other than the rectangularwave and the trigonometric function as shown in FIGS. 7 and 8.Optionally, Tc1, Tc2, Td1 and Td2 may also be set so as to totally cutoff light falling within a particular part of the visible radiationwavelength range. Nevertheless, in order to use the incoming light moreefficiently, the smaller the wavelength range to be cut off, the better,and the higher the overall transmittance, the better.

According to such an arrangement, the light that has entered this imagecapture device during an exposure process passes through thelight-transmitting plate 2, the lens 3, the infrared cut filter 4 andthe transmitting filters 110 and then is incident on the photosensitivecells 120. Each of those photosensitive cells receives a light ray thathas been transmitted through the area C1 or C2 of the light-transmittingplate 2 and then through its associated transmitting filter, and outputsa photoelectrically converted signal representing the quantity of thelight received. The photoelectrically converted signal that has beenoutput from each photosensitive cell is sent to the signal processingsection 200 by way of the signal generating and receiving section 5. Inthe signal processing section 200, the image signal generating section 7generates multi-viewpoint images based on the signals supplied from theimage capturing section 100.

Hereinafter, the photoelectrically converted signals supplied from thosephotosensitive cells will be described. Signals representing therespective intensities of light rays that have been transmitted throughthe areas C1 and C2 and then incident on two pixels of interest will beidentified herein by Ci1 and Ci2, respectively, in a situation where thetransmitting areas C1 and C2 and the transmitting filters D1 and D2 aresupposed to have a transmittance of 100% with respect to any wavelength.Also, a light ray with the same intensity is supposed to be incident oneach of the photosensitive cells included in a single unit element andevery incoming light is supposed to be visible radiation. Furthermore,for the sake of simplicity, the wavelength dependences of theintensities of the light rays that are incident on the areas C1 and C2are neglected. That is to say, the subject is supposed to be in anachromatic color. Also, the spectral transmittance of the lens 3 and theinfrared cut filter 4 combined will be identified herein by Tw. And thespectral transmittances of the areas C1 and C2 will be identified hereinby Tc1 and Tc2, respectively. In the same way, the spectraltransmittances of the transmitting filters D1 and D2 at the image sensor1 will be identified herein by Td1 and Td2, respectively.

In this case, Tw, Tc1, Tc2, Td1 and Td2 are functions that depend on thewavelength λ of the incoming light, and will be represented as Tw(λ),Tc1(λ), Tc2(λ), Td1(λ) and Td2(λ), respectively. And the signalsrepresenting the intensities of light rays that have been transmittedthrough the transmitting filters D1 and D2 and then received byphotosensitive cells that face them are identified by d1 and d2,respectively. Furthermore, the integration operation of the spectraltransmittances in the visible radiation wavelength range will beidentified herein by the sign Σ. For example, an integration operation∫Tw(λ)Tc1(λ)Td1(λ)dλ with respect to the wavelength λ will be identifiedherein by Σ TwTc1Td1. In this case, the integration is supposed to beperformed in the entire visible radiation wavelength range. Then, d1 isproportional to the sum of Ci1 Σ TwTc1Td1 and Ci2 Σ TwTc2Td2. Likewise,d2 is proportional to the sum of Ci1 Σ TwTc1Td2 and Ci2 Σ TwTc2Td2.Supposing the constant of proportionality with respect to theserelations is one, d1 and d2 can be represented by the followingEquations (1) and (2), respectively:d1=Ci1ΣTwTc1Td1+Ci2ΣTwTc2Td1  (1)d2=Ci1ΣTwTc1Td2+Ci2ΣTwTc2Td2  (2)

Suppose, in Equations (1) and (2), Σ TwTc1Td1, Σ TwTc2Td1, Σ TwTc1Td2,and Σ TwTc2Td2 are identified by Mx11, Mx12, Mx21 and Mx22,respectively. Then, Equation (1) can be represented by the followingEquation (3) using a matrix:

$\begin{matrix}{\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix} = {\begin{pmatrix}{{Mx}\; 11} & {{Mx}\; 12} \\{{Mx}\; 21} & {{Mx}\; 22}\end{pmatrix}\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix}}} & (3)\end{matrix}$

Supposing the respective elements of an inverse matrix, which isobtained by inverting the matrix consisting of the elements Mx11 throughMx22 as represented by Equation (3), are identified by iM11 throughiM22, respectively, Equation (3) can be modified into the followingEquation (4). That is to say, the signals Ci1 and Ci2 representing theintensities of the light rays that have been incident on the areas C1and C2 can be represented by using the photoelectrically convertedsignals d1 and d2:

$\begin{matrix}{\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix} = {\begin{pmatrix}{{i{Mx}}\; 11} & {{i{Mx}}\; 12} \\{{i{Mx}}\; 21} & {{i{Mx}}\; 22}\end{pmatrix}\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix}}} & (4)\end{matrix}$

By performing calculations based on this Equation (4), the signals d1and d2 (observed pixel values) representing the quantities of light raysthat have been incident on respective pixels can be converted into thesignals Ci1 and Ci2 representing the intensities of light rays to beincident on the transmitting areas C1 and C2. The image signalgenerating section 7 shown in FIG. 1 carries out a signal arithmeticoperation based on this Equation (4), thereby generating signals Ci1 andCi2 on a unit element basis. These signals Ci1 and Ci2 that have beengenerated on a unit element basis represent two images that have beenproduced by the light rays that were incident on the transmitting areasC1 and C2, respectively. That is to say, these two images formmulti-viewpoint images that have parallax corresponding to the distancebetween the two transmitting areas C1 and C2. Consequently, byperforming the arithmetic operations represented by Equation (4),multi-viewpoint images representing a difference in location between thetransmitting areas C1 and C2 can be generated.

In an actual shooting environment, however, it is difficult to measureaccurately the spectral transmittances of the respective transmittingfilters due to an individual difference between the image sensors 1 orthe light-transmitting plate's (2) transmitting filters or owing to ameasuring error. And if the spectral transmittance characteristic cannotbe measured appropriately, then the respective elements of the matrixrepresented by Equation (3) should have errors. If the errors of therespective elements are identified by Mx11′, MX12′, Mx21′ and Mx22′,respectively, then Equation (3) can be modified into the followingEquation (5):

$\begin{matrix}{\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix} = {\begin{pmatrix}{{{Mx}\; 11} + {{Mx}\; 11^{\prime}}} & {{{Mx}\; 12} + {{Mx}\; 12^{\prime}}} \\{{{Mx}\; 21} + {{Mx}\; 21^{\prime}}} & {{{Mx}\; 22} + {{Mx}\; 22^{\prime}}}\end{pmatrix}\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix}}} & (5)\end{matrix}$

If the inverse matrix of the matrix represented by Equation (5) isobtained and modified into an equation for obtaining Ci1 and Ci2 likeEquation (4), the following Equation (6) is obtained:

$\begin{matrix}{\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix} = {\frac{1}{\det}\begin{pmatrix}{{{Mx}\; 22} + {{Mx}\; 22^{\prime}}} & {{{- {Mx}}\; 12} - {{Mx}\; 12^{\prime}}} \\{{{- {Mx}}\; 21} - {{Mx}\; 21^{\prime}}} & {{{Mx}\; 11} + {{Mx}\; 11^{\prime}}}\end{pmatrix}\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix}}} & (6)\end{matrix}$

In Equation (6), det stands for the determinant of the matrix ofEquation (5) and det=(Mx11+Mx11′)(Mx22+Mx22′)−(Mx12+Mx12′)(Mx21+Mx21′).By expanding Equation (6), Ci1 and Ci2 can be given by the followingEquations (7) and (8), respectively:

$\begin{matrix}{{{Ci}\; 1} = {{\frac{1}{\det}\left( {{d\; 1{Mx}\; 22} - {d\; 2{Mx}\; 12}} \right)} + {\frac{1}{\det}\left( {{d\; 1{Mx}\; 22^{\prime}} - {d\; 2{Mx}\; 12^{\prime}}} \right)}}} & (7) \\{{{Ci}\; 2} = {{\frac{1}{\det}\left( {{d\; 2{Mx}\; 11} - {d\; 1{Mx}\; 21}} \right)} + {\frac{1}{\det}\left( {{d\; 2{Mx}\; 11^{\prime}} - {d\; 1{Mx}\; 21^{\prime}}} \right)}}} & (8)\end{matrix}$

In Equations (7) and (8), errors are involved in only the second term.Thus, these second terms will be referred to herein as “error terms”.These error terms are inversely proportional to det. That is why even ifthe errors are significant but if det is large, the influence of theerrors becomes a limited one. Conversely, even if the errors areinsignificant but if det is small, the influence of the errors becomesfar-reaching. In that case, Ci1 and Ci2 calculated will be quitedifferent from the values obtained when there are no errors.

Also, in an actual shooting environment, since transmitting filters thatattenuate the incoming light are arranged on the light-transmittingplate 2 and the image sensor 1, the quantity of the light received byeach pixel decreases. That is why the image could also involve errorsdue to the influence of thermal noise, for example. In this case, if theerrors that can be involved in the pixel signals d1 and d2 of anobserved image are identified by d1′ and d2′, then Equation (3) can bemodified into the following Equation (9):

$\begin{matrix}{\begin{pmatrix}{{d\; 1} + {d\; 1^{\prime}}} \\{{d\; 2} + {d\; 2^{\prime}}}\end{pmatrix} = {\begin{pmatrix}{{Mx}\; 11} & {{Mx}\; 12} \\{{Mx}\; 21} & {{Mx}\; 22}\end{pmatrix}\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix}}} & (9)\end{matrix}$

If the inverse matrix of the matrix of Equation (9) is obtained andmodified into an equation for obtaining Ci1 and Ci2 just like Equation(4), the following Equation (10) can be obtained:

$\begin{matrix}{\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix} = {\frac{1}{\det}\begin{pmatrix}{{Mx}\; 22} & {{- {Mx}}\; 12} \\{{- {Mx}}\; 21} & {{Mx}\; 11}\end{pmatrix}\begin{pmatrix}{{d\; 1} + {d\; 1^{\prime}}} \\{{d\; 2} + {d\; 2^{\prime}}}\end{pmatrix}}} & (10)\end{matrix}$

In Equation (10), det stands for the determinant of the matrix ofEquation (9) and det=Mx11Mx22−Mx12Mx21. By expanding Equation (10), Ci1and Ci2 can be given by the following Equations (11) and (12),respectively:

$\begin{matrix}{{{Ci}\; 1} = {{\frac{1}{\det}\left( {{d\; 1{Mx}\; 22} - {d\; 2{Mx}\; 12}} \right)} + {\frac{1}{\det}\left( {{d\; 1^{\prime}{Mx}\; 22} - {d\; 2^{\prime}{Mx}\; 12}} \right)}}} & (11) \\{{{Ci}\; 2} = {{\frac{1}{\det}\left( {{d\; 2{Mx}\; 11} - {d\; 1{Mx}\; 21}} \right)} + {\frac{1}{\det}\left( {{d\; 2^{\prime}{Mx}\; 11} - {d\; 1^{\prime}{Mx}\; 21}} \right)}}} & (12)\end{matrix}$

As in Equations (7) and (8), errors are also involved in the second termof Equations (11) and (12). And these error terms are inverselyproportional to det. That is why even if the errors are significant butif det is large, the influence of the errors becomes a limited one.Conversely, even if the errors are insignificant but if det is small,the influence of the errors becomes far-reaching. In that case, Ci1 andCi2 calculated will be quite different from the values obtained whenthere are no errors.

Thus, according to this embodiment, by paying attention to the fact thatthe error terms are inversely proportional to det, the matrix iscorrected so as to increase det, thereby reducing the influence of theerrors. As the determinant det may have a negative value, the matrix iscorrected according to this embodiment so as to increase its absolutevalue |det|. In Equations (7), (8), (11) and (12), the first term isapparently inversely proportional to det. In the processing to bedescribed later, however, the matrix is corrected so that when det isincreased, the numerator of the first term of these equations alsoincreases. Consequently, according to this embodiment, the influence ofonly the error terms can be reduced.

As a method for changing the value of a matrix, a matrix that reducesthe influence of errors by making the respective elements vary freelymay be obtained. If such a method is adopted, the best matrix needs tobe selected from among a huge number of matrices that have beengenerated by changing the values of the respective elements little bylittle, thus imposing a lot of computational load, which is a problem.

Thus, to overcome such a problem, the first and second columns of thematrix of Equation (3) may be represented by vectors A and B,respectively. That is to say, vectors A and B are defined by thefollowing Equation (13):

$\begin{matrix}{{A = \begin{pmatrix}{{Mx}\; 11} \\{{Mx}\; 21}\end{pmatrix}},{B = \begin{pmatrix}{{Mx}\; 12} \\{{Mx}\; 22}\end{pmatrix}}} & (13)\end{matrix}$

FIG. 9 schematically illustrates an example of the vectors A and B.These vectors A and B may be considered as vectors in thetwo-dimensional Euclidean space, and the determinant det may be obtainedby calculating the exterior product of the vectors A and B. That is tosay, the determinant det may be obtained by det=|A∥B|sin θ, where θ isthe angle formed between the vectors A and B. The larger this angle θ,the larger |det|. Conversely, the smaller this angle θ, the smaller|det|.

In view of these considerations, by changing the angle θ from 0 through90 degrees by rotation and by obtaining |det| with respect to each ofthose angles, the best θ can be determined. It should be noted that if θis changed from 0 through 90 degrees, |sin θ| can be changed from itsminimum value of 0 through its maximum value of 1. That is why there isno need to change θ through 360 degrees by rotation.

In this case, |det| becomes maximum when θ=90 degrees (i.e., sin θ=1).Then, the vectors A and B will cross each other at right angles and theinfluence of the error term will be minimum. If the vector B is obtainedby rotating the vector A 90 degrees, then the vector B is given by thefollowing Equation (14):

$\begin{matrix}{B = {{\begin{pmatrix}{\cos\; 90{^\circ}} & {{- \sin}\; 90{^\circ}} \\{\sin\; 90{^\circ}} & {\cos\; 90{^\circ}}\end{pmatrix}\begin{pmatrix}{{Mx}\; 11} \\{{Mx}\; 21}\end{pmatrix}} = \begin{pmatrix}{{- {Mx}}\; 21} \\{{Mx}\; 11}\end{pmatrix}}} & (14)\end{matrix}$

In this example, |A|=|B| is supposed to be met for the sake ofsimplicity. In that case, the vector A becomes (Mx11, Mx21)^(T) and thevector B becomes (−Mx21, Mx11)^(T). Thus, the matrix consisting of thesevectors A and B is represented by the following Equation (15):

$\begin{matrix}{M = \begin{pmatrix}{{Mx}\; 11} & {{- {Mx}}\; 21} \\{{Mx}\; 21} & {{Mx}\; 11}\end{pmatrix}} & (15)\end{matrix}$

By using this matrix M instead of the matrix represented by Equation(3), multi-viewpoint images which are hardly affected by the error termcan be obtained. Specifically, if M represented by Equation (15) isused, Equations (11) and (12) becomeCi1=1/det(d1Mx11+d2Mx21)+1/det(d1′Mx11+d2′Mx21) andCi2=1/det(d2Mx11−d1Mx21)+1/det(d2′Mx11−d1′Mx21), respectively. In suchan image, as the absolute value of Mx21 increases, Ci1 becomes brighterand Ci1 becomes darker (or come to include negative values). A simplemethod to avoid such a situation is to use a filter, of which thespectral transmittance includes Mx21 of zero.

Those equations are satisfied when a light-transmitting plate 2, inwhich red and blue color filters with almost the same transmittance arearranged in the transmitting areas C1 and C2, respectively, and an imagesensor 1, in which red and blue color filters with similartransmittances are arranged as transmitting filters D1 and D2,respectively, are used. In that case, ΣTwTc1Td1=ΣTwTc2Td2 andΣTwTc1Td2=ΣTwTc2Td1=0 are satisfied. In such a situation, however,Ci1=1/det(d1Mx11)+1/det(d1′Mx11) and Ci2=1/det(d2Mx11)+1/det(d2′Mx11)are satisfied and Ci1 and Ci2 are obtained by multiplying the inputsignals d1 and d2 by a constant. That is why even though the influenceof the error term is limited, signals representing the light rays thathave come from the areas C1 and C2 cannot be separated from the pixelsignals. This is because in pixels corresponding to a portion of thesubject, of which the depth does not vary, the pixel signals d1 and d2have substantially the same values. For example, if d1=d2 isapproximately satisfied, two equations corresponding to Equations (11)and (12) when the determinant M is used become the same. Consequently,the signals Ci1 and Ci2 cannot be obtained by making computations. Inthat case, the degree of correlation between the two multi-viewpointimages is so high that the image has almost no parallax, which is aproblem.

Next, it will be described by way of instances what influence the errorterm will have if |det| is small. FIG. 10 shows examples of an inputimage and multi-viewpoint images. Specifically, FIG. 10( a) shows animage (input image) that was obtained by capturing the object ofshooting with a normal color image sensor. FIG. 10( b) showsmulti-viewpoint images that were generated by Equation (4). In thematrix used in this example, the first column was (1, 0.67)^(T), thesecond column was (0.67, 0.5)^(T), and the value of the determinant detwas 0.0511. The image shown on the left-hand side of FIG. 10( b)included negative values in a lot of pixels and was an image of whichthe pixel values were generally small. As can be seen, when such amatrix was used, multi-viewpoint images could not be obtained asintended. Since 1/det≈20 in this example, the error term(d1Mx22′−d2Mx12′) in Equation (7) was amplified twenty-fold. Inaddition, since d1Mx22′<d2Mx12′, the negative error increasedtwenty-fold and multi-viewpoint images could not be obtained asintended. As can be seen, the smaller |det|, the more steeply the errorterm is amplified. As a result, the image generated will have very largepixel values or negative pixel values.

Since these values are outside of the range in which the pixel values ofan image are originally expected to fall, the output image will not be aproper image. That is why the likelihood as an image can be determinedby examining the degree to which the pixel values of multi-viewpointimages generated fall within their originally expected range.

FIG. 11 is a graph showing how the |det| value, the value representingthe correlation between the multi-viewpoint images, and the degree ofconfidence of the multi-viewpoint images change with θ. In this case,the “value representing the correlation between the multi-viewpointimages” is the average of the normalized correlation values that havebeen obtained on an RGB color component basis between the twomulti-viewpoint images. Thus, the closer to one the correlation valueis, the more closely the two images resemble each other. On the otherhand, the “degree of confidence between the multi-viewpoint images”represents the percentage of pixels of each image that fall within theoriginally expected range (e.g., from 0 through 255) of pixel values.

As can be seen from FIG. 11( a), the larger θ, the larger |det|. It canalso be seen that the larger θ, the greater the value of correlationbetween the multi-viewpoint images. And the greater the value ofcorrelation between the multi-viewpoint images, the more closely the twoimages will resemble each other (i.e., the smaller the parallax willbe). In this example, when the normalized correlation value exceeded0.85, the difference between the two images tended to be hardlysensible.

On the other hand, in the example shown in FIG. 11( b), when θ=0.1(rad), one of the two multi-viewpoint images had a degree of confidenceof 98% or more but the other multi-viewpoint image had a degree ofconfidence of approximately 80%. It can be confirmed that as for each ofthese two images, the larger θ, the more significantly the degree ofconfidence increased and the influence of the error term decreased.

To sum up the results of these preliminary experiments, the larger θ,the greater the value correlation between the multi-viewpoint images(i.e., the smaller the parallax) but the higher the percentage ofpixels, of which the pixel values fall within a predetermined range.Thus, it can be seen that it is beneficial to set θ to be a value thatmakes the value of correlation between the multi-viewpoint images fallwithin a range that is smaller than a predetermined value and that makesthe degree of confidence of the multi-viewpoint images fall within arange that is larger than a predetermined value. Thus, according to thisembodiment, the maximum θ is obtained in a range in which the degrees ofconfidence conf(S1, θ) and conf(S2, θ) of the two multi-viewpoint imagesare larger than a predetermined threshold value th and in a range inwhich the value of correlation cor(θ) between the multi-viewpoint imagesis smaller than a predetermined threshold value th′ as represented bythe following Equation (16):

$\begin{matrix}{{{confidence}\left( {S,\theta,x,y} \right)} = \left\{ {{\begin{matrix}{1:{{if}\mspace{14mu}\left( {{{{{0 \leq {S\left( {x,y,\theta} \right)}}\&}\;\&}{S\left( {x,y,\theta} \right)}} \leq 255} \right)}} \\{0:{otherwise}}\end{matrix}{{conf}\left( {S,\theta} \right)}} = {{\sum\limits_{x,y}^{\;}{{{confidence}\left( {S,\theta,x,y} \right)}\theta}} = {\underset{\theta}{\arg\;\max}\left\{ {{{{{{{{{{{conf}\left( {{S\; 1},\theta} \right)} > {th}}\&}\;\&}{{conf}\left( {{S\; 2},\theta} \right)}} > {th}}\&}\;\&}\mspace{14mu}{{cor}(\theta)}} < {th}^{\prime}} \right\}}}} \right.} & (16)\end{matrix}$

In Equation (16), the pixel value at coordinates (x, y) on themulti-viewpoint images with respect to the angle θ is represented asS(x, y, θ). If the value of S(x, y, θ) falls within the range of 0through 255, confidence (S, θ, x, y)=1. Otherwise, confidence (S, θ, x,y)=0. And the sum of confidence (S, θ, x, y) at every set of coordinatesis defined to be the degree of confidence conf(S, θ). That is to say,conf(S, θ) represents the number of pixels, of which the pixel valuesfall within the range of 0 through 255, among all pixels. In a range inwhich the degree of confidence conf(S1, θ) of one of the multi-viewpointimages and the degree of confidence conf(S2, θ) of the othermulti-viewpoint image are both larger than the threshold value th and inwhich the correlation value cor(θ) is smaller than the threshold valueth′, the maximum θ is determined. The threshold values th and th′ may beset to be appropriate values according to the image obtained. In thiscase, the correlation value cor(θ) indicates a normalized correlationbetween two multi-viewpoint images as represented by the followingEquation (17). However, as will be described later, any index other thanthe normalized correlation may also be used as a correlation value.

$\begin{matrix}{{{cor}(\theta)} = \frac{\sum\limits_{x,y}^{\;}{S\; 1\left( {x,y,\theta} \right)S\; 2\left( {x,y,\theta} \right)}}{\sum\limits_{x,y}^{\;}{S\; 1\left( {x,y,\theta} \right)^{2}{\sum\limits_{x,y}^{\;}{S\; 2\left( {x,y,\theta} \right)^{2}}}}}} & (17)\end{matrix}$

FIG. 12 shows an exemplary flow of processing steps to be carried out torealize the processing described above. It should be noted that as theinitial value of the determinant is determined when the spectraltransmittance characteristic is measured, the initial value θ=sin⁻¹(det/|A∥B|) of the parameter θ is also determined in advance. Asdescribed above, the first and second columns of the matrix are supposedto be vectors A and B, respectively, and det is supposed to be anon-zero determinant. First of all, in Step S001, shooting processing iscarried out using the image capture device of this embodiment. Next, inthe processing step S002 of generating multi-viewpoint images,multi-viewpoint images are generated by Equation (4). Subsequently, inthe processing step S003 of calculating a correlation value, a valuerepresenting correlation between the multi-viewpoint images iscalculated. As the correlation value, either the normalized correlationdescribed above or any other index may be used. For example, any ofordinary distance calculating methods or degree of similaritycalculating methods, which use the sum of absolute differences (L1 norm,city block distance) or the sum of squared differences (L2 norm,Euclidean distance) between pixels, for example, may be used. With amethod for calculating the distance based on the difference between theimages adopted, if its inverse number is used as the correlation value,the more closely the two multi-viewpoint images resemble each other, thegreater the value of correlation can be.

In the flow shown in FIG. 12, when the correlation value exceeds thepredetermined threshold value th′ or when the angle of rotation of thevector exceeds 90 degrees, the processing is ended. Otherwise, in theprocessing step S004 of adjusting the matrix, θ is increased so thatθ=θ+step, thereby changing the first column of the matrix. In this case,the angle defined between the vector A consisting of the elements of thefirst column of the matrix and the vector B consisting of the elementsof the second column of the matrix just needs to be increased step bystep, and the vector to rotate does not always have to be the vector A.Alternatively, the vector B may be rotated step by step. Stillalternatively, both of the vectors A and B may be rotated by ±step/2each time.

Examples of multi-viewpoint images generated as a result of theprocessing described above are shown in FIG. 13. Specifically, FIG. 13shows multi-viewpoint images that were newly generated based on theinput image shown in FIG. 10(a) by using a matrix that was corrected byincreasing θ. In FIG. 13, the vertically running white line passesthrough the same x coordinate on the two images. As indicated by thewhite line, the paper crane's head is shifted to the right in the upperimage compared to the lower image. These results proved theeffectiveness of the method of this embodiment empirically, too.

By performing the processing described above, even if matrix elements orpixel signals involve errors, quality multi-viewpoint images withparallax can still be generated. Considering the errors of pixelsignals, the image quality can also be improved even in a dark sceneincluding a lot of noise. Also, in the transmitting areas C1 and C2 ofthe light-transmitting plate 2 and in the transmitting filters D1 and D2of the image sensor 1, if a lot of light is transmitted, then therespective elements of the matrix will have mutually close values, |det|will decrease, and therefore, the image quality will be affected by theerrors more easily. Even so, according to this embodiment, the influenceof such errors can also be reduced so much that filters with hightransmittance, with which images can be shot with high sensitivity, canbe used, which is beneficial.

In the embodiment described above, the determinant is supposed to beincreased by gradually increasing θ. However, this is just an exampleand the determinant may also be increased by any other method. Since thedeterminant det is represented by |A| |B|sin θ, det increases as θincreases in the range in which 0≦θ≦90 degrees. That is why a new matrixmay be obtained by changing det into a predetermined relatively largevalue det′, obtaining θ′ associated with det′ by θ′=sin⁻¹(det′/(|A∥B|)),and then rotating at least one of the vectors A and B so that the anglebecomes equal to or greater than θ′. In this case, however, det issupposed to be changed so that the angle obtained based on the initialvalue det satisfies θ+90 degrees>θ′. As it can be seen what value thedeterminant needs to be changed into in order to reduce the influence ofthe errors sufficiently in an environment where the magnitude of theerrors can be seen, θ′ can be obtained by directly specifying det′ evenwithout performing the repetitive processing.

In the foregoing description, the matrix adjustment processing issupposed to be performed every time an image is shot. However, thematrix adjustment processing does not have to be performed every timeshooting is done. For example, if the transmitting areas C1 and C2 ofthe light-transmitting plate 2 and the transmitting filters D1 and D2 ofthe image sensor 1 do not change, the transmittance measuring error doesnot change, either. That is why by using a matrix obtained with θ thathas been estimated in advance by the method of this embodiment,multi-viewpoint images with little error can also be generated even foran image representing a different scene. Also, in shooting a movingpicture, the transmitting areas C1 and C2 and the transmitting filtersD1 and D2 do not change and scenes that change continuously need to beshot. That is why the processing of obtaining θ from one image frame ofthe moving picture may be performed and then multi-viewpoint images maybe calculated by using θ that has already been obtained for the framesthat follow. As a result, the computational load of the θ estimationprocessing while shooting a moving picture can be lightened.

Embodiment 2

Hereinafter, a second embodiment of the present invention will bedescribed.

In the image capture device of the first embodiment described above, thelight-transmitting plate 2 has two transmitting filters with mutuallydifferent spectral transmittance characteristics, so does each unitelement of the image sensor 1. However, the present invention is in noway limited to that specific embodiment. The light-transmitting plate 2and each unit element of the image sensor 1 may each have three or moretransmitting filters or may have mutually different numbers oftransmitting filters. Hereinafter, a generalized one of theconfiguration of the first embodiment, in which m (where m is an integerthat is equal to or greater than two) transmitting filters are arrangedin the light-transmitting plate 2 and in which n (where n is an integerthat is equal to or greater than m) transmitting filters are providedfor each unit element of the image sensor 1, will be described. Theimage capture device of this embodiment is quite the same as the firstembodiment described above except the configurations of thelight-transmitting plate 2 and the image sensor 1 and the processingperformed by the image signal generating section 7. The followingdescription of this second embodiment will be focused on thosedifferences from the first embodiment and their common features will notbe described all over again to avoid redundancies.

FIG. 14 schematically illustrates an exemplary configuration for thelight-transmitting plate 2 of this embodiment. The light-transmittingplate 2 of this embodiment has m transmitting areas C1, C2, . . . andCm, in each of which a transmitting filter is arranged. Thetransmittances of these m transmitting areas C1 through Cm have mutuallydifferent wavelength dependences. The rest of the light-transmittingplate 2 other than the m transmitting filters is an opaque area thatdoes not transmit light. In FIG. 14, all of those transmitting areas aredrawn as circular ones with the same planar area. However, the shape andsize of the respective transmitting areas do not have to be theillustrated ones. The arrangement of the respective transmitting areasdoes not have to the illustrated one, either, but any other arrangementmay be adopted as well. Likewise, the light-transmitting plate 2 doesnot have to have a circular shape, either but may also be aquadrilateral or any other shape. Furthermore, although the lighttransmitting plate 2 has an opaque area according to this embodiment,the opaque area may also be made of a light-transmitting member andtreated as a transmitting area, too.

FIG. 15 schematically illustrates an exemplary arrangement of ntransmitting filters that are included in each unit element 40 of theimage sensor 1 of this embodiment. Each unit element 40 of the imagesensor 1 includes n photosensitive cells and n transmitting filters thatface them. The transmittances of these n transmitting filters D1, D2, .. . and Dn have mutually different wavelength dependences. It should benoted that the arrangement shown in FIG. 15 is just an example andpixels may also be arranged in any other pattern within each unitelement 40.

Suppose, in the configuration described above, the pixel signals outputfrom the transmitting filters D1, D2, . . . and Dn of the image sensor 1are identified by d1, d2, . . . and dn, respectively, and signalsrepresenting the intensities of light rays that are incident on therespective photosensitive cells from the transmitting areas C1, C2, . .. and Cm in a situation where the transmittances of the transmittingareas C1 through Cm and the transmitting filters D1 through Dn aresupposed to be 100% are identified by Ci1, Ci2, . . . and Cim. In thatcase, the relation between the pixel signals d1, d2, . . . and dn andthe image signals Ci1, Ci2, . . . and Cim is represented by thefollowing Equation (18):

$\begin{matrix}{\begin{pmatrix}{d\; 1} \\{d\; 2} \\\vdots \\{dn}\end{pmatrix} = {\begin{pmatrix}{{Mx}\; 1\; 1} & {{Mx}\; 12} & \ldots & {{Mx}\; 1m} \\{{Mx}\; 2\; 1} & {{Mx}\; 22} & \ldots & {{Mx}\; 2m} \\\vdots & \vdots & \ddots & \vdots \\{{Mxn}\; 1} & {{Mxn}\; 2} & \ldots & {Mxnm}\end{pmatrix}\begin{pmatrix}{{Ci}\; 1} \\{Ci2} \\\vdots \\{Cim}\end{pmatrix}}} & (18)\end{matrix}$

In the first embodiment described above, the 2×2 matrix is transformedinto a matrix that is not affected by errors easily. In this embodiment,on the other hand, the n×m matrix represented by Equation (18) istransformed by a similar method into a matrix that is not affected byerrors easily. In the first embodiment, a method of making the anglebetween the two vectors closer to 90 degrees (i.e., a method ofincreasing the angle between the two vectors) is adopted in order toincrease |det|. If this method is extended to the n×m matrix, the bestmatrix may be estimated by using m vectors consisting of the elements ofeach column of the matrix. Specifically, the angles between the vectorsare obtained from among _(m)C₂ combinations, each of which is defined bychoosing two out of m vectors. Since a combination in which the anglebetween the vectors is the smallest is a combination of vectors thatwill make calculation of multi-viewpoint images least stabilized, amatrix is newly obtained by increasing the angle between those vectorsby the same method as what is adopted in the first embodiment describedabove. That is to say, with the angle between the vectors increased stepby step, the degree of confidence and the correlation value are obtainedand the degree of likelihood of the multi-viewpoint images generated isdetermined. As a result, even if the number of vectors is m, the bestmatrix can be obtained on a step by step basis and multi-viewpointimages can also be generated.

Next, suppose what if only two arbitrary ones of the m multi-viewpointimages should have their errors reduced during generation. In that case,by choosing two vectors from the matrix and increasing the angle θbetween the vectors, only two arbitrary ones of the m multi-viewpointimages can have their errors reduced. This method is also applicable toa situation where only k arbitrary ones (where k is an integer that isequal to or greater than three) of the m multi-viewpoint images shouldhave their errors reduced. As a result, among multiple images generatedby imaging the light that has come through the light transmittingsection with multiple transmitting areas, only images associated withselected transmitting areas can have its image quality improved.

The image processing section 7 of this embodiment generates mmulti-viewpoint images associated with m transmitting areas byperforming the processing described above. However, the image processingsection 7 may also generate images associated with only some of the mtransmitting areas. For example, if only horizontal parallax informationneeds to be obtained, the image processing section 7 may be configuredto generate images based on the light beams that have been incident ononly two transmitting areas that are horizontally spaced apart from eachother among those m transmitting areas.

The image capture device of the first and second embodiments describedabove generates an image signal by performing signal arithmeticoperations on a photoelectrically converted signal that has beenobtained by capturing an image. However, such processing of generatingan image signal by performing signal arithmetic operations may also becarried out by another device that is provided independently of thatimage capture device. For example, even if a signal that has beenobtained by an image capture device including the image capturingsection 100 of this embodiment is loaded into another device (imageprocessor) to get a program defining the signal arithmetic processingdescribed above executed by a computer in that another device, theeffects of the embodiments described above can also be achieved.

INDUSTRIAL APPLICABILITY

A 3D image capture device according to an embodiment of the presentinvention can be used effectively in any camera that ever uses an imagesensor. Examples of those cameras include consumer electronic camerassuch as digital still cameras and digital camcorders and solid-statesurveillance cameras for industrial use.

REFERENCE SIGNS LIST

-   1 solid-state image sensor-   1 a imaging area of solid-state image sensor-   2 light-transmitting plate-   3 optical lens-   3 a optical element functioning as both light-transmitting plate and    optical lens-   4 infrared cut filter-   5 signal generating and receiving section-   6 sensor driving section-   7 image signal generating section-   8 interface section-   19 lens diaphragm-   20, 22, 23 light beam confining plate-   20 a color filter transmitting red-based light ray-   20 b color filter transmitting blue-based light ray-   21 photosensitive film-   22R, 23R R ray transmitting areas of light beam confining plate-   22G, 23G G ray transmitting areas of light beam confining plate-   22B, 23B B ray transmitting areas of light beam confining plate-   30 memory-   40 unit element-   100 image capturing section-   110 transmitting filter-   120 photosensitive cell-   200 signal processing section

The invention claimed is:
 1. A 3D image capture device comprising: alight transmitting section with m transmitting areas (where m is aninteger that is equal to or greater than two) that have mutuallydifferent spectral transmittance characteristics; an image sensor whichis arranged to receive light rays that have been transmitted through them transmitting areas and in which a plurality of unit elements arearranged, each said unit element including n photosensitive cells (wheren is an integer that is equal to or greater than m) and n transmittingfilters with mutually different spectral transmittance characteristicswhich are arranged so as to face the n photosensitive cells; an imagingsection which produces an image on the imaging area of the image sensor;and an image processing section which modifies an n×m matrix that isdefined by the respective spectral transmittance characteristics of them transmitting areas and the n transmitting filters by changing theangle between at least two out of m column vectors of the n×m matrix andwhich generates multi-viewpoint images represented by light rays thathave been incident on at least two of the m transmitting areas based onthe modified n×m matrix and n photoelectrically converted signalssupplied from the n photosensitive cells.
 2. The 3D image capture deviceof claim 1, wherein the image processing section increases thedeterminant of the n×m matrix by increasing the angle between at leasttwo of the m column vectors.
 3. The 3D image capture device of claim 1,wherein the image processing section changes the angle so that a valuerepresenting correlation between the multi-viewpoint images becomessmaller than a predetermined threshold value.
 4. The 3D image capturedevice of claim 1, wherein the image processing section increases theangle so that the determinant of the n×m matrix becomes greater than apredetermined threshold value.
 5. The 3D image capture device of claim1, wherein the image processing section increases the angle between twocolumn vectors that is smaller than the angle formed by any other pairof the m column vectors.
 6. The 3D image capture device of claim 1,wherein the image processing section obtains the angle between twoarbitrary ones of the m column vectors and increases the anglessequentially by beginning with the smallest one, thereby increasing thedeterminant of the n×m matrix.
 7. The 3D image capture device of claim1, wherein m=2 and n=2.
 8. An image processor which generatesmulti-viewpoint images based on a signal that has been obtained by a 3Dimage capture device, the device comprising: a light transmittingsection with m transmitting areas (where m is an integer that is equalto or greater than two) that have mutually different spectraltransmittance characteristics; an image sensor which is arranged toreceive light rays that have been transmitted through the m transmittingareas and in which a plurality of unit elements are arranged, each saidunit element including n photosensitive cells (where n is an integerthat is equal to or greater than m) and n transmitting filters withmutually different spectral transmittance characteristics which arearranged so as to face the n photosensitive cells; and an imagingsection which produces an image on the imaging area of the image sensor,wherein the image processor modifies an n×m matrix that is defined bythe respective spectral transmittance characteristics of the mtransmitting areas and the n transmitting filters by changing the anglebetween at least two out of m column vectors of the n×m matrix andgenerates multi-viewpoint images represented by light rays that havebeen incident on at least two of the m transmitting areas based on themodified n×m matrix and n photoelectrically converted signals suppliedfrom the n photosensitive cells.
 9. An image processing method forgenerating multi-viewpoint images based on a signal that has beenobtained by a 3D image capture device, the device comprising: a lighttransmitting section with m transmitting areas (where m is an integerthat is equal to or greater than two) that have mutually differentspectral transmittance characteristics; an image sensor which isarranged to receive light rays that have been transmitted through the mtransmitting areas and in which a plurality of unit elements arearranged, each said unit element including n photosensitive cells (wheren is an integer that is equal to or greater than m) and n transmittingfilters with mutually different spectral transmittance characteristicswhich are arranged so as to face the n photosensitive cells; and animaging section which produces an image on the imaging area of the imagesensor, wherein the image processing method comprises the steps of:modifying an n×m matrix that is defined by the respective spectraltransmittance characteristics of the m transmitting areas and the ntransmitting filters by changing the angle between at least two out of mcolumn vectors of the n×m matrix; and generating multi-viewpoint imagesrepresented by light rays that have been incident on at least two of them transmitting areas based on the modified n×m matrix and nphotoelectrically converted signals supplied from the n photosensitivecells.
 10. An image processing program stored on a non-transitorycomputer readable medium for generating multi-viewpoint images based ona signal that has been obtained by a 3D image capture device, the devicecomprising: a light transmitting section with m transmitting areas(where m is an integer that is equal to or greater than two) that havemutually different spectral transmittance characteristics; an imagesensor which is arranged to receive light rays that have beentransmitted through the m transmitting areas and in which a plurality ofunit elements are arranged, each said unit element including nphotosensitive cells (where n is an integer that is equal to or greaterthan m) and n transmitting filters with mutually different spectraltransmittance characteristics which are arranged so as to face the nphotosensitive cells; and an imaging section which produces an image onthe imaging area of the image sensor, wherein the program is defined soas to make a computer perform the steps of: modifying an n×m matrix thatis defined by the respective spectral transmittance characteristics ofthe m transmitting areas and the n transmitting filters by changing theangle between at least two out of m column vectors of the n×m matrix;and generating multi-viewpoint images represented by light rays thathave been incident on at least two of the m transmitting areas based onthe modified n×m matrix and n photoelectrically converted signalssupplied from the n photosensitive cells.